Изменения
→Байесовская классификация
Sbt зависимость:
libraryDependencies '''+= ''' "org.apache.spark" '''%% ''' "spark-core" '''% ''' "2.4.0" libraryDependencies '''+= ''' "org.apache.spark" '''%% ''' "spark-mllib" '''% ''' "2.4.0" '''% ''' "runtime"
Пример линейной регрессии c применением org.apache.spark.ml.regression.LinearRegression<ref>[https://spark.apache.org/docs/latest/ml-classification-regression.html#linear-regression Spark, LinearRegression]</ref>:
'''val '''training = spark.read.format("libsvm")
.load("linear_regression.txt")
'''val '''lr = '''new '''LinearRegression()
.setMaxIter(10)
.setRegParam(0.3)
.setElasticNetParam(0.8)
'''val '''lrModel = lr.fit(training)
Вывод итоговых параметров модели:
println(lrModel.coefficients)
println(lrModel.intercept)
'''val '''trainingSummary = lrModel.summary
println(trainingSummary.totalIterations)
println(trainingSummary.objectiveHistory.mkString(","))
Sbt зависимость:
libraryDependencies '''+= ''' "com.github.haifengl" '''%% ''' "smile-scala" '''% ''' "1.5.2"
Пример ридж и лассо регрессии c применением smile.regression<ref>[https://haifengl.github.io/smile/regression.html Smile, Regression]</ref>:
'''import '''smile.data.{AttributeDataset, NumericAttribute} '''import '''smile.read '''import '''smile.regression.{LASSO, RidgeRegression, lasso, ridge}
'''val '''data: AttributeDataset = read.table("regression.txt", delimiter = " ", response = Some(('''new '''NumericAttribute("class"), 0))) '''val '''x: Array[Array['''Double''']] = data.x() '''val '''y: Array['''Double'''] = data.y() '''val '''ridgeRegression: RidgeRegression = ridge(x, y, 0.0057) '''val '''lassoRegression: LASSO = lasso(x, y, 10)
println(ridgeRegression)
println(lassoRegression)
Sbt зависимость:
libraryDependencies '''+= ''' "org.apache.spark" '''%% ''' "spark-core" '''% ''' "2.4.0" libraryDependencies '''+= ''' "org.apache.spark" '''%% ''' "spark-mllib" '''% ''' "2.4.0" '''% ''' "runtime"
Пример логистической регрессии c применением spark.mllib.classification<ref>[https://spark.apache.org/docs/2.3.1/mllib-linear-methods.html#logistic-regression Spark, Logistic Regression]</ref>:
'''import '''org.apache.spark.mllib.classification.{LogisticRegressionModel, LogisticRegressionWithLBFGS} '''import '''org.apache.spark.mllib.evaluation.MulticlassMetrics '''import '''org.apache.spark.mllib.regression.LabeledPoint '''import '''org.apache.spark.mllib.util.MLUtils
'''val '''data = MLUtils.loadLibSVMFile(sc, "logisticRegresion.txt") '''val '''splits = data.randomSplit(Array(0.6, 0.4), seed = 11L) '''val '''training = splits(0).cache() '''val '''test = splits(1) '''val '''model = '''new '''LogisticRegressionWithLBFGS()
.setNumClasses(10)
.run(training)
'''val '''predictionAndLabels = test.map { '''case '''LabeledPoint(label, features) => '''val '''prediction = model.predict(features)
(prediction, label)
}
'''val '''metrics = '''new '''MulticlassMetrics(predictionAndLabels) '''val '''accuracy = metrics.accuracy
println(accuracy)
Sbt зависимость:
libraryDependencies '''+= ''' "com.github.haifengl" '''%% ''' "smile-scala" '''% ''' "1.5.2"
Пример классификации c применением smile.classification.mlp<ref>[https://haifengl.github.io/smile/classification.html#neural-network Smile, MLP]</ref>:
'''import '''smile.classification.NeuralNetwork.{ActivationFunction, ErrorFunction} '''import '''smile.data.{AttributeDataset, NumericAttribute} '''import '''smile.read '''import '''smile.classification.mlp '''import '''smile.plot.plot
'''val '''data: AttributeDataset = read.table("iris.csv", delimiter = ",", response = Some(('''new '''NumericAttribute("class"), 2))) '''val '''x: Array[Array['''Double''']] = data.x() '''val '''y: Array['''Int'''] = data.y().map(_.toInt) '''val '''mlpModel = mlp(x, y, Array(2, 10, 2), ErrorFunction.LEAST_MEAN_SQUARES, ActivationFunction.LOGISTIC_SIGMOID)
plot(x, y, mlpModel)
Пример кода, с использованием билиотеки DeepLearning.scala
<span style="color:#3D9970>// Задание слоёв</span> '''def '''tanh(x: INDArrayLayer): INDArrayLayer = { '''val '''exp_x = hyperparameters.exp(x) '''val '''exp_nx = hyperparameters.exp(-x)
(exp_x - exp_nx) / (exp_x + exp_nx)
}
'''def '''charRNN(x: INDArray, y: INDArray, hprev: INDArrayLayer): (DoubleLayer, INDArrayLayer, INDArrayLayer) = { '''val '''hnext = tanh(wxh.dot(x) + whh.dot(hprev) + bh) '''val '''yraw = why.dot(hnext) + by '''val '''yraw_exp = hyperparameters.exp(yraw) '''val '''prob = yraw_exp / yraw_exp.sum '''val '''loss = -hyperparameters.log((prob * y).sum)
(loss, prob, hnext)
}
<span style="color:#3D9970>// Определение структуры</span> '''val '''batches = data.zip(data.tail).grouped(seqLength).toVector
type WithHiddenLayer[A] = (A, INDArrayLayer)
type Batch = IndexedSeq[(Char, Char)]
type Losses = Vector['''Double'''] '''def '''singleBatch(batch: WithHiddenLayer[Batch]): WithHiddenLayer[DoubleLayer] = { batch '''match '''{ '''case '''(batchseq, hprev) => batchseq.foldLeft((DoubleLayer(0.0.forward), hprev)) {
(bstate: WithHiddenLayer[DoubleLayer], xy: (Char, Char)) =>
(bstate, xy) '''match '''{ '''case '''((tot, localhprev), (x, y)) => { charRNN(oneOfK(x), oneOfK(y), localhprev) '''match '''{ '''case '''(localloss, _, localhnext) => {
(tot + localloss, localhnext)
}
}
<span style="color:#3D9970>// Определение одного шага обучения</span> '''def '''initH = INDArrayLayer(Nd4j.zeros(hiddenSize, 1).forward) '''def '''singleRound(initprevloss: Losses): Future[Losses] =
(batches.foldLeftM((initprevloss, initH)) {
(bstate: WithHiddenLayer[Losses], batch: Batch) =>
bstate '''match '''{ '''case '''(prevloss, hprev) => singleBatch(batch, hprev) '''match '''{ '''case '''(bloss, hnext) => bloss.train.map { (blossval: '''Double''') => { '''val '''nloss = prevloss.last * 0.999 + blossval * 0.001 '''val '''loss_seq = prevloss :+ prevloss.last * 0.999 + blossval * 0.001
(loss_seq, hnext)
}
}).map {
(fstate: WithHiddenLayer[Losses]) =>
fstate '''match '''{ '''case '''(floss, _) => floss
}
}
'''def '''allRounds: Future[Losses] = (0 until 2048).foldLeftM(Vector(-math.log(1.0 / vocabSize) * seqLength)) { (ploss: Losses, round: '''Int''') => {
singleRound(ploss)
}
}
<span style="color:#3D9970>// Обучение сети</span> '''def '''unsafePerformFuture[A](f: Future[A]): A = Await.result(f.toScalaFuture, Duration.Inf) '''val '''losses = unsafePerformFuture(allRounds)
===Долгая краткосрочная память===
SBT зависимость:
libraryDependencies '''+= ''' "com.github.haifengl" '''%% ''' "smile-scala" '''% ''' "1.5.2"
Пример классификации датасета и вычисления F1 меры<ref>[https://en.wikipedia.org/wiki/F1_score F1 мера]</ref> используя smile.classification.svm<ref>[https://haifengl.github.io/smile/classification.html#svm Smile, SVM]</ref>:
'''import '''smile.classification._ '''import '''smile.data._ '''import '''smile.plot._ '''import '''smile.read '''import '''smile.validation.FMeasure
'''val '''iris: AttributeDataset = read.table("iris.csv", delimiter = ",", response = Some(('''new '''NumericAttribute("class"), 2))) '''val '''x: Array[Array['''Double''']] = iris.x() '''val '''y: Array['''Int'''] = iris.y().map(_.toInt) '''val '''SVM = svm(x, y, '''new '''GaussianKernel(8.0), 100) '''val '''predictions: Array['''Int'''] = x.map(SVM.predict) '''val '''f1Score = '''new '''FMeasure().measure(predictions, y)
plot(x, y, SVM)
===Дерево решений и случайный лес===
Освновная статья: [[Дерево решений и случайный лес#Пример использования на языке Scala | Дерево решений и случайный лес: Пример использования на языке Scala]].
SBT зависимость:
libraryDependencies '''+= ''' "com.tsukaby" '''%% ''' "naive-bayes-classifier-scala" '''% ''' "0.2.0"
Пример классификации используя smile.classification.cart<ref>[https://github.com/tsukaby/naive-bayes-classifier-scala Naive bayes classifier, Scala]</ref>:
<span style="color:#3D9970>// Создание модели</span> '''val '''bayes = '''new '''BayesClassifier[String, String]() <span style="color:#3D9970>// Задание соотвествия категория - слово</span>
bayes.learn("technology", "github" :: "git" :: "tech" :: "technology" :: Nil)
bayes.learn("weather", "sun" :: "rain" :: "cloud" :: "weather" :: "snow" :: Nil)
bayes.learn("government", "ballot" :: "winner" :: "party" :: "money" :: "candidate" :: Nil)
<span style="color:#3D9970>// Тестовые примеры</span> '''val '''unknownText1 = "I use git".split(" ") '''val '''unknownText2 = "Today's weather is snow".split(" ") '''val '''unknownText3 = "I will vote '''for '''that party".split(" ") <span style="color:#3D9970>// Классификация</span> println(bayes.classify(unknownText1).map(_.category).getOrElse("")) <span style="color:#3D9970>// technology</span> println(bayes.classify(unknownText2).map(_.category).getOrElse("")) <span style="color:#3D9970>// weather</span> println(bayes.classify(unknownText3).map(_.category).getOrElse("")) <span style="color:#3D9970>// government</span>
===EM-алгоритм===
SBT зависимость:
libraryDependencies '''+= ''' "com.tsukabygithub.haifengl" '''%% ''' "naive-bayes-classifiersmile-scala" '''% ''' "01.5.2.0"
Пример классификации используя smile.clustering.kmeans<ref>[https://haifengl.github.io/smile/clustering.html#k-means Smile, K-Means]</ref>:
'''import '''smile.clustering._ '''import '''smile.data._ '''import '''smile.plot._ '''import '''smile.read
'''val '''iris: AttributeDataset = read.table("iris.csv", delimiter = ",", response = Some(('''new '''NumericAttribute("class"), 2))) '''val '''x: Array[Array['''Double''']] = iris.x() '''val '''kMeans: KMeans = kmeans(x, k = 6, maxIter = 1000) '''val '''y = kMeans.getClusterLabel
plot(x, y, '.', Palette.COLORS)
